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Abstract 

In this era of external teacher testing with the intent of ensuring the 
competence of the teaching force, as well as holding students and institutions 
accountable for results, the Idaho Comprehensive Dteracy Assessment (ICLA) 
stands in sharp contrast. It represents an alternative to external testing of pre- 
service teachers, testing procured from an outside agency unconnected to pre- 
service programs, since it has been developed and is managed by literacy 
faculty from Idaho’s major institutions of higher learning. This paper provides 
a brief history of major events in the field of literacy including teacher testing 
initiatives and policies, which led to the creation of the ICLA. A description of 
the ICLA assessment and its construction is provided, along with a report of 
initial performance. Implications and policy consequences of this approach are 
explored. 
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Regardless of what one thinks of the poUcy, teacher testing has arrived at the national and 
State policy scene and will likely remain for the foreseeable future (Ludlow, Shirley, and 
Rosea, 2002). The No Child Left Behind Act of 2001 (Public Law 107-110) mandates 
competent teachers in all classrooms and that prospective teachers demonstrate this 
competence before entering the teaching force. States have logically moved to establish 
teacher tests to determine the competence of pre-service teachers. In addition, with the 
reauthorization of Tide 11 of the Higher Education Act in 1998 (Public Law 105-244), 
teacher education programs must annually report the performance of pre-service teachers on 
a variety of state tests. Teacher testing has become a mainstay in teacher education and 
certification. 

Youngs, Odden, and Porter (2003) provide a snapshot of state pre-service testing 
practices for the 2001-2002 academic year, showing the extent to which teacher testing has 
expanded in a relatively short time. Thirty-seven of 50 states employ basic skills tests for 
licensure, 33 states require tests in content knowledge, and 26 states require pedagogy tests. 
Given the momentum of this poUcy across states, and national legislation authorizing its 
implementation, the number of states with teacher testing policies is undoubtedly growing. 

Despite the momentum for teacher testing, the policy is not without justifiable 
criticism. As an example, Ludlow (2001) identified sub-standard psychometric characteristics 
of teacher tests in Alabama and Massachusetts. For Alabama, the consequence of poor test 
development practice was severe. The court directed Alabama to pay large compensation 
packages to individuals who showed that poor test development adversely impacted many 
test takers, and that in some cases, the test discriminated against minority groups. 

Teacher testing in Massachusetts received considerable attention. The highly 
politicized and publicized teacher testing resulted in firestorm of controversy (Flippo, & 
Riccards, 2000; Haney, Fowler, & Wheelock, 1999; Ludlow, Shirley, and Rosea, 2002). 
Apparently, the content of the test is unconnected to good teaching or what teachers actually 
do (Ludlow, Shirley, & Rosea). In addition, although the Massachusetts test is lauded as a 
mastery test, norm-referenced test development procedures were used, undermining the 
mastery intent of the test (Haney, 1999). The end result is the mis-measurement of teacher 
competence in Massachusetts. The inflammatory language used by State policy makers to 
describe those who failed the test and institutions that performed poorly is disappointing 
and illustrates the challenges ahead as states continue to impose teacher testing. 

It is clear that the intent of these teacher tests is to hold students and the system 
accountable. The accountability approach relies upon an external test, a test developed by an 
external agency, that requires certain pass scores for students and pass rates for institutions. 

One real consequence is the possibility of public shaming of teacher education 
programs that do not meet pass-rate expectations; another is sanctions, which threaten the 
very existence of the programs. In response, for example, many institutions in Massachusetts 
conduct a variety of program initiatives in an attempt to boost pass rates (Ludlow, Shirley, & 
Rosea, 2002). Some of the initiatives have actually worked. However, it is unclear whether 
higher pass rates on tests considered unconnected to teaching, indicate that more competent 
teachers are entering the teaching force. 
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A Focus on Reading 

While discussions and debates about the role and place of teacher testing continue, 
other issues in teacher education have also emerged. One such issue is reading instruction. 
The focus on literacy occurred as a natural outcome of numerous investigations and reports 
detailing poor reading achievement in the United States and proposals for intervention. 
Correspondingly, teacher preparation in reading has been subject to considerable debate and 
discussion. 

A resonant theme from the debate about reading is that too many students cannot 
meet basic literacy standards. Numerous reports have called for reform. The first of these 
reports came in 1983 when the National Commission on Excellence in Education reported 
to the Secretary of the Department of Education and to the nation that student achievement 
in reading, math, and science had declined since. In the well known report tided, A Nation 
At Risk, the Nadonal Commission on Excellence in Education claimed that students in the 
United States experienced shorter school days, shorter school years, less homework, and 
lower graduation requirements than did students in many other developed nations. They 
urged the Nation to set higher educational standards and provide the resources that would 
assure that the United States would remain globally competitive. 

In 1990, then President Bush and the Nation’s governors established six national 
educational goals for the year 2000. Goals 4 and 6 called for literacy to become a top priority 
for all students K-12 (Executive Office of the President, 1990). Under President Clinton, the 
number of goals was expanded to eight (Alliance 2000 Project, 1994), including the focus on 
reading. 

In 1997 Congress asked the Director of the National Institute of Child Health and 
Human Development, in consultation with the Secretary of Education, to form a panel to 
synthesize research on reading instruction. This National Reading Panel (2000) analyzed 
research in the areas of alphabetics, fluency, comprehension, teacher education and reading 
instruction, and computer technology and reading instmction. The report was controversial, 
further complicating the discussion and debate about reading and how best to achieve 
reading outcomes. Moreover, serious questions were raised about the trustworthiness of the 
report itself. Yatvin (2002), a member of the panel, was one of the panel’s critics. She relates 
a disturbing narrative of events that culminated in the final report being published without 
adequate proofing. 

CamiUi, Vargas, and Yurecko (2003) replicated the National Reading Panel’s meta- 
analysis of phonics instmction. After arriving at conclusions different from those in the 
report, they raised questions regarding National Reading Panel’s methodology and the link 
between the evidence and the Panel’s conclusions. 

Recent No Child Left Behind legislation (Public Law 107-110, 2001) reiterated 
recommendations from previous reports. In addition, it requires that every teacher be highly 
qualified in the subjects taught, including reading. Paraprofessionals would need to earn the 
equivalent of an Associate Arts Degree. Close monitoring of the progress of individual 
students would become necessary in order to identify potential at risk learners, intervene 
instructionally, and measure the effects of instruction on an ongoing basis. The need to 
interpret these assessments and respond instructionally highlighted the importance of 
teachers and paraprofessionals having an in depth knowledge of beginning reading 
instruction and the decoding process. 
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A second strand in the debate surrounding reading instruction comes from 
researchers and educators in literacy education. In response to the National Commission on 
Excellence in Education (1983) report,^ Nation at Task, the National Academy of 
Education Commission on Reading (1985) published becoming a Nation of Readers. This report 
stated that reading is a complex process and not a series of isolated sub-skills. This report 
was cautious about the recommendations from the National Commission on Excellence in 
Education report. Becoming a Nation of Readers reasserted that teachers have a greater impact 
on literacy learning than any instructional approach, set of materials, or structural change, a 
recurring theme stiU advanced today. 

In 1990, Adams published the seminal work Beginning to Read: Thinking and Teaming 
about Trint. This extensive review of the research on reading supported the recommendations 
in Becoming A Nation of Readers (1985). Systematic phonics instruction, while essential, should 
be accompanied by reading quality children’s literature. In addition, as prerequisite skills for 
phonics, phonemic and phonological awareness are essential to early success in reading. 

Eight years later. Snow, Burns and Griffin (1998) edited Preventing Reading Difficulties in 
Young Children, which summarized an analysis of 30 years of scientific research on reading 
instruction and causes of reading failure in young children. Recommendations from this 
review mirrored those of Adams (1990) by calling for systematic instruction in reading along 
with wide reading and writing both in and out of school. The position taken was that such 
instruction benefits not only students at risk in reading, but all students as well. 

A related body of research has examined the effects of teacher decision-making on 
reading achievement. Findings have consistendy shown that teachers who make good 
decisions regarding instruction are more effective than any single instmctional strategy 
deemed best practice (Duffy & Hoffman, 1999; Pearson, 1997; RuddeU, 1997). Good 
instructional decisions coupled with good pedagogical strategies are the most effective ways 
to teach children to read. Duffy and Hoffman call for more flexibility in public educational 
policies that value teachers’ decision making. 

Because of the growing interest in teacher education as a critical component in 
systemic reform efforts, particularly as it pertains to reading instruction, the International 
Reading Association established the National Commission on Excellence in Elementary 
Teacher Preparation for Reading Instmction, commonly known as the IRA Commission 
(Maloch, Fine, & Flint, 2002). The IRA Commission identified teacher education program 
features that lead to excellent reading instruction and student reading achievement. These 
features include: 

A clearly articulated institutional mission 

Committed faculty 

♦J* High standards and multiple measures used for selection, monitoring, and support 

Strong emphasis on current literacy theory and best practices 
♦J* Faculty modeling instmctional elements they want to see in their students. 

♦♦♦ Well supervised field experiences 
♦J* The fostering of professional identity 

Faculty exercising autonomy 

An interesting first year finding by the IRA Commission was that teachers who had 
graduated from one of these programs responded to their students’ needs in flexible, 
knowledgeable, and strategic ways. In point of fact, these are the types of teachers that 
research has demonstrated are the most effective (Duffy & Hoffman, 1999; RuddeU, 1997). 



Education Volicy Analysis Archives Vol 12 No. 61 


5 


In sum, numerous reports have documented the perceived and real need for 
increased student reading achievement and enhanced teacher preparation. These reports 
have called for changes in the time allocated to literacy instruction, curriculum development, 
instructional strategies and materials, allocation of human resources for remedial instruction, 
and an increased use of both summative and formative assessments. Because there is 
considerable research that suggests teacher decision-making, along with sound instmctional 
approaches, are likely the most important factors impacting student reading achievement, 
teacher preparation programs have reexamined the content and pedagogical knowledge 
emphasized in their literacy courses. There has been a substantial shift in focus, 
consequendy. Phonological, phonic and stmctural analysis knowledge, accompanied by 
renewed support for direct, systematic approaches for teaching beginning reading, 
characterize many reading methods courses required of pre-service teachers today. 

The brief historical review here is not meant to be comprehensive. It is illustrative of 
the division between policy makers and educators. Policy makers focus on stmctural change 
and the use of particular reading strategies to improve reading. Educators cite research 
showing that teacher decision making, along with good instmctional strategies, provide the 
best hope for increased reading achievement. Idaho embraced the latter set of ideas as the 
State sought to improve reading achievement. 


Idaho Comprehensive Literacy Initiative 

With the national focus on literacy intensifying, Idaho, like many other States, began 
to examine in greater depth the reading performance of its K-8 smdents. A study of 938 
fourth grade students in 41 schools across Idaho found that, depending upon which of six 
common reading measures were employed, from 21% to a high of 62% of all students tested 
read below grade level (Canney, 1998; 1999). Of equal concern was the finding that among 
smdents qualifying for special assistance (Reading Recovery, Tide I, Limited English 
Proficiency, and Migrant), from 42% to 100% of the smdents assessed were not reading on a 
fourth grade level, depending upon the group and the assessment measure. These findings 
motivated the development of legislation affecting the content and focus of reading 
instruction K-8. Teams of teachers, administrators, university faculty, and interested citizens 
convened to develop standards for reading instmction and recommendations for best 
practice (Idaho’s MOST, 2001). 

Seeing pre-service education as a key component in systemic State education reform 
policy, the Idaho Legislamre also called for the creation of the Idaho Comprehensive 
Literacy Assessment (Idaho Stamte 33-1207A, 1999), which pre-service elementary and 
special education teachers would have to pass to certify after August 2002. To recertify, K-8 
teachers and principals would have to either pass the same assessment or successfully 
complete a three-credit course — the Idaho Comprehensive Literacy Course. 

Efforts to assess teacher knowledge of best practices in reading had already begun in 
two other States. In California, pre-service teachers applying for a teaching credential after 
September 1998 need to pass the Reading Instmction Competence Assessment (RICA), 
available in a written form or in a video performance form. Four areas are targeted: 
instruction based assessment; phonological and linguistic processes; comprehension; oral 
and written language development. In the written form candidates examine acmal student 
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artifacts to write long-range instructional plans (Carlson, 1999; California Commission on 
Teacher CredentiaUng, 1998). 

Texas began its pre-service teacher-testing program in 1986. The most recent version 
is a set of assessments known as Texas Examinations of Educator Standards (TExES). The 
TExES is made up of 33 separate assessments that test a variety of content areas and 
specialties (State Board of Educator Certification, 2003). There are four English Language 
Arts/Reading assessments within this set of 33 assessments. While TExES remains under 
development, each K-12 pre-service teacher must pass two TExES assessments, one on 
pedagogy, the second in the pre-service teacher’s area of expertise (M. Janysek, personal 
communication, September 11, 2003). 

Interest in Idaho for measuring pre-service teachers’ literacy knowledge and 
performance was informed by the Standards for Professionals in Reading developed by the 
International Reading Association, Professional Standards and Ethics Committee (1998). 

The document contains 1 6 knowledge, performance, and disposition standards for reading 
professionals. The Standards became an important guide for the committee developing core 
standards for initial teacher preparation in the language arts. The Standards also influenced 
the Idaho Comprehensive Literacy Assessment committee’s efforts to create an assessment 
that included both a measure of pre-service teachers’ knowledge of research-based best 
practices in reading, and their ability to apply this information to classroom teaching 
situations. The Idaho Comprehensive Literacy Assessment committee made a deliberate 
move to develop an assessment that reflects the knowledge and performance 
recommendations of the IRA Commission and what research in reading instmction has 
shown to be most effective. 


The Idaho Comprehensive Literacy Assessment 

At the direction of the State Board of Education, a committee of literacy professors, 
classroom teachers, school administrators, and State Department representatives formed to 
develop the Idaho Comprehensive Literacy Assessment (ICLA). After some deliberation, a 
committee of literacy professors at various institutions in the State worked to create the 
ICLA, resulting in a pool of items assembled as two forms of the ICLA. Three Statewide 
pilot tests of these forms were conducted with pre-service elementary education majors 
between April and December of 2001. 

Measurement consultants conducted statistical analyses for item and test 
development. The percent correct for an item was used as an initial means to consider the 
quality of an item. Since the ICLA is a criterion-referenced test, the percent correct was used 
not as means to discriminate among students (as done with norm-referenced tests), but used 
instead as an indicator of mastery (Anastasi & Urbina, 1997). Operationally, if more than 
50% of the students responding to an item answered incorrectly, the item was examined by 
the committee to determine if the item was flawed, needed revision, or should remain. This 
process continued systematically after each pilot administration and remains the current 
practice. 

The constmction of the ICLA was guided by the development of a validity 
framework (Shepard, 1993). The framework specified the rationale for the assessment, the 
conceptual basis for the assessment, and logical questions that could be asked of the 
assessment and process. In addition, the Office of Civil Rights (Berkowitz, Wolkowitz, 

Fitch, & Kopriva, 2000) developed a set of recommendations for the development of high 
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stakes tests to ensure that the test does not unfairly disadvantage students from under- 
represented groups. This guide also informed the construction of the ICLA. 

Format 

The Idaho Comprehensive Literacy Assessment is divided into three Standards. 
Standard I, Language Learning and Literacy Development, addresses emergent literacy, 
phonological and phonemic awareness, phonics and structural analysis, sight vocabulary, 
morphemic analysis, and research-based instructional practices for developing accurate and 
automatic decoding. Standard II, Reading Comprehension Research and Best Practices, 
focuses on fluency, vocabulary development, comprehension instruction, and text genres. 
Standard III, Dteracy Assessment and Intervention, deals with common assessment 
procedures, interpretation of assessment results, and instructional activities for stmggUng 
readers. 

Each Standard, in turn, is separated into three sections — 1, 2, and 3. Sections 1 and 2 
assess basic knowledge of key vocabulary, reading skills, and instmctional strategies and 
procedures. All items are objectively scored. Section 3 contains two open-ended essay 
questions that assess students’ application of knowledge; each essay question depicts a 
classroom scenario. Students are asked to explain the purpose of the instructional activity, 
provide a rationale in light of current research, and give two or three alternative activities 
that address the same instructional objective. 

A third version of the assessment now exists, and a fourth version is under 
development. In addition, an on-line version of the assessment has been piloted, but no 
study comparing the paper and pencil and on-line assessment has been conducted. Pass rates 
appear to be comparable. Due to lack of resources, refinement of the on-line version has 
been put on hold. 

To assist students preparing for the ICLA, an electronic study guide (Squires, 
Blacklock, Christy, Nelson, and Palmer, 2002) was created with content information and 
sample test items. This guide is available at no charge to students. In addition, each 
institution schedules test review sessions prior to the administration of the next round of the 
ICLA. 


Scoring Rubric and Protocol 

The ICLA Committee created a four-point mbric with which to score the scenarios. 
The rubric identifies how well a respondent addresses a scenario question, including 
references to research-based best practices, evidence of understanding and sensitivity to 
learners’ differences, and the rationale behind teachers’ instmctional decisions. 

Scenario responses are scored by ICLA committee members, all of whom are literacy 
professors at the several teacher preparation institutions (4 public, 3 private) across the State. 
Scoring the essay items takes place at a central location. If pre-service teachers respond with 
unexpected information, the scenario question is reviewed and, if needed, revised for clarity 
in subsequent administrations. Before any scenario is scored, the mbric is reviewed and 
important elements of smdent responses discussed. For new members, this procedure serves 
as an effective induction process. 
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Two evaluators read each response. If their two scores are identical or differ by only 
one point, the score is averaged between the two readings. If the score of the two readers 
differs by more than one point, a third evaluator, without knowledge of the previous two 
scores, also rates the response. Should two evaluators agree on a score, that is the score 
assigned. If the three evaluators disagree, the score is averaged. 

Setting Pass Scores 

Standard-setting processes were employed to determine the overall pass score for the 
ICLA. After some deliberation, the ICLA committee decided that each of the three 
assessment Standards should receive a separate pass score. A measurement consultant 
guided the standard-setting process by first providing the committee with the advantages and 
disadvantages of various standard-setting approaches. Given its simplicity and popularity, the 
committee decided to use the Angoff Method for the objective items; the Benchmark 
method was used for the scenarios (Cizek, 1996). 

The process resulted in setting a 70% pass score for each Standard and each form. In 
addition, the committee considered the relatively heavy influence of the objective items on 
the pass score. Initially, a student could pass a Standard without obtaining any points on the 
scenarios. The committee selected a 60-40 percentage split between objective items and 
scenarios. To accomplish this split, weights were applied to the scenarios in each Standard. 

Fairness Review 

A fairness review committee was established to investigate, identify, and correct any 
item thought to disadvantage students based on race, gender, ethnicity, or disability (Kimtta 
& Goellner, 2002). The fairness review committee was composed of eight individuals 
representing people of diverse groups. Representation included the disabled, English as 
Second Language, African American, Native American, Latin American, lower socio- 
economic status, and gender. 

The fairness review committee was provided with a reading of the review process, a 
rubric, and a matrix recording form. During the review, members of the committee read 
each test item and flagged the items that posed a problem of bias. Each problem item was 
brought before the committee for discussion. The results of this work were then discussed 
with the ICLA Committee. Items of concern were corrected by changes in wording, 
inserting geographic examples from Idaho, or proper name changes. 

Administration Procedures 

The ICLA is administered three times per year, fall, spring, and summer. Pre-service 
teachers may take one or more Standards during an assessment period, but only one 
Standard per day, and no Standard more than once during the assessment period. Students 
failing to pass a Standard may take another form of that Standard at the next testing session. 
Students unable to pass one or more Standards after three attempts may request a special 
oral examination. Accommodations are available for special needs students. 
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Summary of Initial Assessment Results 

The following provides a summary of the initial ICLA results for Idaho pre-service 
teachers by Standard and administration period. Pass rates by attempt are also provided, as 
well as knowledge (sections 1 and 2 objective items) versus performance (section 3 
scenarios) results. 

Total Pass Rates by Standard 

Table 1 presents the overall pass rates by Standard for the initial four administration 
periods, beginning in April 2002. Standard I, Language Learning and Literacy Development, 
consistendy produced the lowest pass rates. There was an overall increase across testing 
periods in the percentage of pre-service teachers passing Standard I, with a high of 72 
percent passing in December 2002. The number of pre-service teachers passing dropped to 
66 percent in April 2003. The pass rates for Standards II and III remained relatively high in 
comparison to Standard I performance. This pattern was consistent across administrations. 

Table 1 

Total ICLA Pass Rates by Standard, April 2002 - April 2003 


Standard 

Number Taking 

Number Passing 

Percent Passing 

1 

1,457 

934 

64 

2 

850 

713 

84 

3 

838 

712 

85 


Pass Rates by Attempt 

In April 2002, 186 of the 326 examinees taking one or more of Standards I-III were 
doing so for the first time. The other examinees had exposure to one or more of the pilot 
administrations. April 2002 was also the first fully operational administration of the ICLA. 
Thus, the results of the first time test takers, starting with the first administration, were 
tracked to determine the number of attempts needed to pass a particular standard. The data 
provided baseline information to the Idaho teacher preparation institutions. 

Standard I. Table 2 provides the cumulative pass rates for Standards I, II and III. In 
April 2002, 80 of 186 pre-service teachers (43 percent) passed Standard I on their first 
attempt. The following July, 45 of 61 pre-service teachers (24 percent of the 186 pre-service 
teachers starting in April 2002) passed on their second trial. In December 2002, 16 of 31 pre- 
service teachers taking the Standard for the third time passed (9 percent of 186 pre-service 
teachers starting in April 2002). In April 2003, 12 of 14 making a fourth attempt passed (6 
percent of 186). Altogether, of the 186 pre-service teachers who took Standard I in April 
2002 for the first time, 153 passed (82 percent). 
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Table 2 

Cumulative Pass Rates for Candidates taking Standards I, II, and/ or III one or 
more times between April 2002 - April 2003 


Standard 

Number Taking 

Number Passing Percent Passing 

Cumulative Percent 

1 

186 

153 

82% 

43-82% 

2 

138 

136 

98% 

93-98% 

3 

144 

129 

90% 

85-90% 


Standard II. Using the same cumulative analysis for Standard 11, after three 
attempts beginning in April 2002, and ending in December 2002, the cumulative pass rate 
increased from 93% to 98%. 

Standard III. For Standard III, from the first to the fourth administration, the 
cumulative pass rate increased from 85% to 90%. 


Discussion 

The ICLA has now been operational for approximately two years, after seven 
administrations of the test. During the beginning stages of the ICLA development, the 
committee set a Statewide and institutional goal of an 80% pass rate for each Standard. Some 
institutions have reached this goal and, while the goal has been met for Standards II and III 
across the State, students continue to find Standard I to be the most difficult. In addition, 
students continue to perform better on the knowledge portion of the assessment (Sections 1 
and 2) than on the performance part of the test. These two issues — Standard I difficulty and 
knowledge versus performance scores, continue to garner a good deal of attention by the 
committee. Ways to improve the assessment, changes to the curriculum, and revisions to 
instruction have been topics of ongoing discussion. 

The authors acknowledge the controversy surrounding teacher testing and the 
limited research that connects pre-service training and testing with student achievement. Our 
argument, however, is that given mandated testing, there are far more benefits to using the 
approach discussed in this paper, than could be obtained through the widespread practice of 
using externally procured tests. We also recognize that there have been reservations about 
using faculty-controUed assessments. A common argument is that the faculty-controUed 
assessments and scoring process lack objectivity. Faculty members are viewed as being too 
close to day-to-day instmction and, as a result, are too vested in the pass rate outcome to 
remain unbiased. Oversight, therefore, must come from the outside. However, our 
experience with what has actually occurred in Idaho when higher education faculty members 
work in a coUegial and professional manner has shown that concerns about faculty bias have 
been unfounded. To the contrary, the benefits of this collaboration have been impressive. 
Although faculty members set an 80% pass rate, this target has yet to be achieved for all 
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cohorts and standards Statewide. In addition, some institutions have stmggled to meet the 
80% target, even after multiple attempts by individual students. Faculty members have 
consistendy shown themselves unafraid to impose a rigorous examination, even if it means 
some students may fail or that their respective institutions may not meet the 80% target. 

Still others may argue that faculty members across institutions who espouse different 
philosophical orientations about how students learn to read and therefore how reading 
should be taught, are ill-suited to productively work together to create and sustain an 
assessment of this magnitude. Again, the evidence suggests otherwise. Literacy faculty 
members from institutions in Idaho that often compete with one another for enrollment, 
resources, and political support, have chosen to work together amicably. Faculty members 
have also worked to overcome differences of opinion about best instructional practices. This 
level of cooperation was accomplished by first agreeing on the outcomes assessed by the 
ICLA. Once this understanding was reached, faculty members agreed to support the 
possibility of multiple strategies for obtaining literacy learning outcomes for Idaho pre- 
service teachers. Thus, phonics and whole language approaches remain firmly embedded at 
the institutional level. 

The ICLA committee has worked professionally and responsibly to implement 
operational policies concerning the ICLA. These policies are always developed with 
forethought and fairness in mind. Policies include when and how the ICLA will be 
administered, number of possible retakes, accommodations, and the like. Recently, the State 
Board has charged the Colleges of Education Deans’ Council to oversee the work of the 
ICLA committee and to develop a budget for the ongoing work of the committee. The 
Deans’ Council has remained supportive of the work of the faculty and the role of the ICLA. 
In large part, faculty members remain in control of the assessment. 

What, then, does this ICLA experiment offer the teacher education community, as 
well as State and Federal policy makers? First, it is a fundamentally different test 
development and refinement process than the externally contracted tests now sweeping the 
country. While the ICLA is in direct response to State legislation focused on improving 
reading instmction, the process embraced in Idaho is one under the purview of faculty 
members directly responsible for literacy courses in pre-service programs in the State. 
Second, it is an assessment approach decidedly weighted in favor of program improvement 
rather than common notions of accountability through external testing. Finally, it is a 
process that exemplifies the recommendations put forth by the National Commission on 
Excellence in Elementary Teacher Preparation for Reading Instruction (Maloch, Fine, & 
Flint, 2002). 

The test is definitely high stakes for students and institutions. Students who do not 
pass the test cannot be credentialed in the State of Idaho. Institutions are faced with 
ensuring that a high percentage of their graduates can demonstrate content and pedagogical 
knowledge appropriate for well-prepared teachers (Public Law 107-110, 2001). However, 
what the State receives for reformulating current notions of accountability is a test with 
relevance. Rather than external teacher tests with a lack of connection to what teachers 
should know and do, this Idaho test assesses content and performance outcomes directly 
related to State and National professional standards for pre-service literacy preparation. 
Moreover, the ICLA was developed with rigorous measurement procedures that meet 
industry standards. These measurement procedures are consistent with the criterion- 
referenced nature of the ICLA. The ICLA data indicate that the test provides reliable and 
valid results that can be used to make sound decisions about teacher competence and 
refinements to pre-service teacher preparation programs. 
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A further benefit of this testing process is its contribution to the continuous 
improvement of pre-service programs. Faculty members develop tests, arrange for their 
administration, analy 2 e the results by item and task, and discuss results. Since faculty 
members are intimately involved in all aspects of the assessment process, they act on the 
results (Weiss, 1998). Based on regular discussions of test results and accompanying input 
from colleagues across the State, faculty members make changes in their pre-service course 
curricula and instruction. Thus, this approach stands in direct contrast to external testing 
strategies by external agencies which, to date, have not been shown to effect changes in pre- 
service courses (Cochran-Smith & Dudley-MarUng, 2001). 

Despite the ICLA’s initial accomplishments, repeated attempts to obtain a budget 
from the State for this project have failed. The funding to initiate the project came from 
one-time State allocations for a related literacy initiative. However, these funds have not 
been replenished. The funds for the current operation are derived from modest student 
registration fees, while the faculty members continue to invest enormous time without 
compensation. The work to print, disseminate, score, and send reports to institutions, is 
accomplished through short-term time slip employees. Clearly, the infrastmcture supporting 
the ICLA is fragile and its future is in question. 

We want to underscore the fact that the cost for a testing approach such as the ICLA 
is likely a fraction of what states typically spend on externally procured teacher tests. By 
employing a testing process like the ICLA, states will not only have a test with relevance 
used, in part, to improve pre-service instruction, they will also have a testing process with 
reduced expenditures on teacher testing, saving scarce state resources. 

Two additional challenges for the ICLA will need attention. First, much of the policy 
rhetoric concerning teacher testing is focused on the assessment of performance. A handful 
of states have explored the implementation of actual performance assessments, although 
because of the cost and difficulty of establishing adequate reliability and validity, most states 
have shied away from these assessments (Youngs, Odden, & Porter, 2003). Early 
deliberations concerning the ICLA stressed the importance of assessment of performance, 
but given the resources the choice was made to opt for more cost effective and efficient 
assessment strategies. A consequence is the use of less authentic approaches to assessing 
performance outcomes, but the push for assessment of performance remains. The 
committee has outlined a new version of the ICLA that would make it more performance- 
oriented than the current paper- and-pencil version; development awaits funding. 

Second, given the high stakes nature of teacher tests, concern for disadvantaged pre- 
service teacher candidates will continue to grow (Young, Odden, & Porter, 2003). The high 
stakes testing guidelines developed by the Federal government (U. S. Department of 
Education, 2000) as well as the Standards for Educational and Psychological Testing (American 
Educational Research Association; American Psychological Association; National Council on 
Measurement in Education; Joint Committee on Standards for Educational and 
Psychological Testing, 1999) provide the legal, professional, and moral imperative for 
monitoring differential performance across student groups. This monitoring is a critical 
aspect of the ICLA process. Currendy, the number of students in any historically 
disadvantaged student group is so small in Idaho that stadstical analysis has been rendered 
meaningless. A list of acceptable accommodadons for special needs students is now part of 
the policy governing the ICLA and is clearly oudined in the ICLA Study Guide. In addition, 
given the relatively small number of students in teacher education programs in general, 
faculty members know students personally, particularly students from underrepresented 
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groups. Careful oversight for the welfare of these students has been occurring at the 
program and institutional level. The ICLA Committee wiU continue to attend to this issue. 


Summary 

As external teacher testing continues to work its way into State and Federal reform 
policy, the ICLA stands apart. By using an assessment developed and maintained by faculty, 
rather than one developed, procured, and maintained by an external agency, the ICLA 
represents another way for State and Federal policy makers to think about teacher testing 
and the benefits and limitations associated with such practices. The ICLA test and process is 
decidedly focused on rigorously assessing pre-service teacher knowledge and performance in 
reading, and using the results to revise pre-service instruction. No external teacher test has 
shown to impact the curriculum and instruction of pre-service instructional programs. 
However, in Idaho the effect of pre-service teacher performance on the ICLA, particularly 
on Standard 1, has directly impacted the literacy course content and delivery at Idaho teacher 
preparation programs. 
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